Welcome to Apache Airflow Overview. After watching this video, you will be able
to: Recognize Apache Airflow as a platform to
programmatically author, schedule, and monitor workflows. List the main features and principles of Apache
Airflow. And, list common use cases for Apache Airflow. Apache Airflow is a great open source workflow
orchestration tool that is supported by an active community. It is a platform that lets you build and run
workflows, such as batch data pipelines. With Apache Airflow, a workflow is represented
as a DAG (a Directed Acyclic Graph), and contains individual pieces of work called tasks, arranged
with dependencies. And, note that unlike Big Data tools such
as Apache Kafka, Apache Storm, Apache Spark, or Flink, Apache Airflow is not a data streaming
solution. It is primarily a workflow manager. Let’s take a look at a simplified overview
of Apache Airflow’s basic components. Airflow comes with a built-in Scheduler, which
handles the triggering of all scheduled workflows. The Scheduler is responsible for submitting
individual tasks from each scheduled workflow to the Executor. The Executor handles the running of these
tasks by assigning them to Workers, which then run the tasks. The Web Server , serves Airflow’s powerful
interactive User Interface. From this UI, you can inspect, trigger and
debug any of your DAGs and their individual tasks. The DAG Directory contains all of your DAG
files, ready to be accessed by the Scheduler, the Executor, and each of its employed Workers. Finally, Airflow hosts a Metadata Database,
which is used by the Scheduler, Executor, and the Web Server to store the state of each
DAG and its tasks. A DAG specifies the dependencies between tasks,
and the order in which to execute them. The tasks themselves describe what to do. In this
example DAG, the tasks include data ingestion, data analysis, saving the data, generating
reports, and triggering other systems, such as reporting any errors by email. Let’s have a look at the life cycle of a
task’s state. In this diagram, you can see how Apache Airflow
might assign states to a task during its life cycle. No status: This means that the task has not
yet been queued for execution. Scheduled: The scheduler has determined that
the task’s dependencies are met and has scheduled it to run. Removed: For some reason, the task has vanished
from the DAG since the run started. Upstream failed: An upstream task has failed. Queued: The task has been assigned to the
Executor and is waiting for a worker to become available. Running: The task is being run by a worker. Success: The task finished running without
errors. Failed: The task had an error during execution
and failed to run, and Up for retry: The task failed but has retry
attempts left and will be rescheduled. Ideally, a task should flow through the scheduler
from ‘no status’, to ‘scheduled’, to ‘queued’, to ‘running’, and finally
to ‘success.’ Now, let’s have a look at the five main
features and benefits of Apache Airflow. Pure Python Create your workflows using standard Python.
This allows you to maintain full flexibility when building your data pipelines. Useful UI Monitor, schedule, and manage your workflows
via a sophisticated web app, offering you full insight into the status of your tasks. Integration Apache Airflow provides many plug-and-play
integrations, such as IBM Cloudant, that are ready to execute your tasks. Easy to Use Anyone with Python knowledge can deploy a
workflow. Airflow does not limit the scope of your pipelines. And finally, the open source feature. Whenever you want to share your improvement,
you can do this by opening a pull request. Airflow has many active users who are sharing
their experiences in the Apache Airflow community. Apache Airflow pipelines are built on four
main principles. They are: Scalable: Airflow has a modular architecture
and uses a message queue to orchestrate an arbitrary number of workers. It is ready to
scale to infinity. Dynamic: Airflow pipelines are defined in
Python, and allow dynamic pipeline generation. Thus, your pipelines can contain multiple
simultaneous tasks. Extensible: You can easily define your own
operators and extend libraries to suit your environment. And, Lean: Airflow pipelines are lean and
explicit. Parameterization is built into its core using the powerful Jinja templating engine. Apache Airflow has supported many companies
in reaching their goals. For example, Sift used Airflow for defining and organizing
Machine Learning pipeline dependencies, SeniorLink increased the visibility of their
batch processes and decoupled them, Experity deployed Airflow as an enterprise
scheduling tool, and Onefootball used Airflow to orchestrate SQL
transformations in their data warehouses, and to send daily analytics emails. In this video, you learned that: Apache Airflow is a platform to programmatically
author, schedule, and monitor workflows. The five main features of Airflow are its
use of Python, its intuitive and useful user interface, extensive plug-and-play integrations,
ease of use, and the fact that it is open source. You also learned that Apache Airflow is scalable,
dynamic, extensible, and lean. And finally, defining and organizing machine
learning pipeline dependencies with Apache Airflow is one of the common use cases.